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Abstract 

This paper describes aspects of the design and implementation of ‘Learning How to Learn - in 
classrooms, schools and networks’, a major development and research project within the UK’s 
Teaching and Learning Research Programme. It focuses particularly on how a group of 
Cambridge academics and researchers, working in partnership with other universities, school 
districts and schools, is attempting to meet the legitimate demands of these different 
communities for different kinds of knowledge. First we describe the overall design of the project, 
then we look at one element of it in more detail: the design and use of a questionnaire for school 
staff and how the results are used in feedback for development. Finally, we summarise some of 
the challenges that arise. 
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' This paper is based on the work of ‘Learning How to Learn - in classrooms, schools and 
networks'. This is a four year development and research project funded from January 2001 to 
March 2005 by the UK Economic and Social Research Council (ref: LI 39 25 1020) as part of 
Phase II of the Teaching and Learning Research Programme (see http://www.tlrp.org ). The 
Project is directed by Mary James (University of Cambridge) and co-directed by Robert 
McCormick (Open University). Other members of the team are: Carmel Burgess, Patrick 
Carmichael, David Frost, John MacBeath, David Pedder and Sue Swaffield (University of 
Cambridge), Paul Black, Bethan Marshall and Joanna Swann (King's College London), Leslie 
Honour and Richard Procter (University of Reading) and Alison Fox (Open University). Past 
members of the team are Geoff Southworth, University of Reading (until March 2002), Coliii 
Conner, University of Cambridge (until April 2003) and Dylan Wiliam, King's College London 
(until August 2003). Further details are available at: 
http://www.leamtoleam.ac.uk . 
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Introduction 



Although Cambridge is perceived as one of the most academic universities in the UK, the 
Faculty of Education has a proud tradition of working practically with teachers, schools and 
school districts to promote better learning for students and teachers. In part this derived from the 
incorporation into the University of the former Institute of Education which was originally set up 
by the national Goverranent to support in-service education of teachers in the East Anglian 
region of England. Over time those academics working within this tradition came not only to 
ground their work on evidence from research but also to contribute significantly to the 
knowledge base (see, for example, McCormick and James, 1983: Bradley, Conner and 
Southworth, 1994; James, 1998; Frost et al., 2000; MacBeath and Mortimore, 2001). With 
incorporation into an elite university and the pressures from successive rounds of a ‘high stakes’ 
national research assessment exercise, academics have sometimes experienced a tension between 
the desire to do research which has high relevance to practitioners and policy-makers and the 
aspiration to do research which is high quality in scientific terms as judged through peer review. 

Working in Pasteur’s Quadrant (Stokes, 1997), although a laudable aim, is not easily achievable 
away from the laboratory in the messy world of schools, although it is undoubtedly something 
worth striving for. Thus, the “Learning How to Learn - in classrooms, schools and networks” 
(L2L) Project is an attempt to design a collaborative development and research initiative with the 
twin goals of achieving high relevance and high scientific quality^. In essence this was an 
expectation placed upon the project through participation in the UK Economic and Social 
Research Council’s Teaching and Learning Research Programme (TLRP) - the largest 
programme of research in education that has ever been funded in the UK. When Phase III of 
TL^ is launched in June 2003, more than 30 major projects will be supported under the 
Programme^. Each is expected to: 

• work to achieve significant improvements in learning outcomes for identified groups of 
learners; 

• work in authentic settings of teaching and learning; 

• bring multi-disciplinary or interdisciplinary approaches to research, and, where appropriate, 
involve practitioners, learners and other potential beneficiaries in the research process; 

• enhance the capacity for a research-based approach to education and training practices; 

• work in partnership with practitioners, learners, policy makers and others in the research 
community, to achieve maximum impact through transformation of the research results into 
actionable strategies and practices; 

• make research-based contributions to the fundamental understanding of teaching and 
learning. 

For L2L this means that the project team is required to work with, and respond to, equally, the 
concerns of the communities of scholars and the communities of education policy and practice - 
to serve at least two masters. 



^ ‘Scientific’ is understood here in a broad sense, not narrowly restricted to experiments, but nevertheless striving to 
meet the kind of criteria set out by the US National Research Council Committee (NRC, 2002). 

^ See httD://tlrD,org ^ 



The development and research design 
The origins of an idea 

L2L is a four year project (Jan 2001 -March 2005) involving four universities (Cambridge 
University, King’s College London, the Open University, and Reading University) working in 
partnership with 43 elementary and high schools in seven school districts (Essex, Hertfordshire, 
Kent, Medway, Oxfordshire, Redbridge and Somerset). The idea for the project came from two 
sources, both intending to build on previous work of the researchers in the different institutions. 

The first source was work on assessment for learning. James (Cambridge) was convenor of the 
UK’s Assessment Reform Group (ARG)" at the time when it commissioned from Black and 
Wiliam (King’s London) a review of research of the impact of assessment on classroom learning, 
in order to up date aspects of an earlier review by Crooks (1988). This new review (Black and 
Wiliam, 1998a), which paid particular attention to fairly carefully controlled, small scale, 
focused experiments, reported that there was conclusive evidence that formative assessment 
(now more familiarly referred to in the UK as ‘assessment for learning’) improves learning and 
attainment. Indeed, the gains in attainment appeared to be quite considerable and, with effect 
sizes from 0.4 to 0.7, amongst the largest ever reported for educational interventions. The 
dissemination of this review through publication in a user-friendly pamphlet (Black and Wiliam, 

1 998b), talks given to teachers by researchers, and follow-up publications from the Assessment 
Reform Group (ARG, 1 999, 2002), contributed to an unusual level of take up of ideas by 
practitioners and policy groups. For example, the Government’s recommended strategy for 
teachers of 1 1 - 14 year old students (ref: DfES 0350/2002) includes a module on ‘assessment for 
learning in everyday lessons’ that derives from this work. The Qualifications and Curriculum 
Authority has also dedicated a section of its website to this theme^ . 

Black and Wiliam subsequently undertook a development project involving 24 math and science 
teachers in six high schools in two school districts. This concentrated on promoting new 
practices in four aspects of effective assessment for learning: developing classroom dialogue, 
especially questioning; using narrative comments, rather than scores, in feedback on students’ 
work; peer- and self-assessment; and, the formative use of summative tests. The results of this 
work indicate that teachers’ and students’ expectations, and classroom culture, can be changed 
and that these changes are associated with gains in attainment (in this case the average effect size 
was around 0.3). The researchers claim that, ‘Such improvements, produced across a school, 
would raise a school in the lower quartile of the national performance tables to well above 
average’ (Black, et al., 2002:4). Here’s the rub! How does one spread knowledge and promote 
changes in practice across teachers and schools? Moreover, how can one achieve ‘leverage’ 
using minimum resource for maximum impact? These questions engage with a second field of 
educational research, associated with school improvement and, more generally, with educational 
change. 



■' See http://assessment-reform-grouD.org.uk 
* See http://www.Qca.org.uk/ca/5- 1 4/afl/ 
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As mentioned earlier, colleagues at Cambridge University have worked in the field of teacher 
professional development and school improvement for many years, both as educators and 
researchers. Much has been learned about ways to create the conditions for school improvement 
and there has been a discernible shift from an earlier emphasis on school structures and staff 
roles and responsibilities, which are at one remove from the primary processes of schooling, 
towards a more explicit focus on conditions and processes that support effective teaching and 
learning. For example, MacBeath and Mortimore (2001:21) argue that critical for school 
effectiveness and school improvement is a better understanding of the when and the how of 
learning. This may seem to be stating the obvious but it is remarkable how infrequently school 
improvement researchers have engaged with issues of pedagogy (and vice versa). These two 
fields of classroom-level research and school-level research are vast in themselves and bringing 
them into alignment may seem too difficult. However, unless some attempt is made to 
understand what effective learning looks like, what teaching practices promote this, and what 
kinds of professional development and institutional conditions and cultures help teachers to learn 
new practices, we have little chance of spreading and sustaining the kinds of improvements 
observed in the studies reviewed by Black and Wiliam. 

The L2L Project was therefore set up to respond to this challenge: to take forward the classroom 
work begun by Black and Wiliam and link it with the school improvement work of Cambridge 
colleagues. Moreover, in anticipation of schools becoming only one site for learning in the 
electronic age, as the creation of dispersed but networked learning communities increases, the 
research interests of other colleagues from Reading University and the Open University, who are 
especially interested in electronic media for networking, were drawn in. This enabled a three 
level project to be designed which would seek to investigate what learning how to learn might 
mean for teachers and students in classrooms, in relation to organisational learning, and for 
learning across networks of teachers and schools. Specifically, the project aims to advance both 
understanding and practice of learning how to learn in classrooms (level 1), schools (level 2) and 
networks (level 3) and to: 

1 . develop and extend recent work on ‘assessment for learning’ into a model of learning how to 
learn for both teachers and pupils; 

2. investigate what teachers can do to help pupils to learn how to learn; 

3. investigate what characterises the school in which teachers successfully create and manage 
the knowledge and skills of learning how to learn; 

4. investigate how educational networks, including electronic networks, can support the 
creation, management and transfer of the knowledge and skills of learning how to learn; 

5. attempt to develop a generic model of innovation in teaching and learning that integrates 
work in classrooms, schools and networks. 

Design assumptions and principles 

The project design was based on a number of theoretical, methodological or practical 
assumptions: 

• The concepts and practices underpinning ‘assessment for learning’ cohere theoretically 
with concepts and practices associated with ‘learning how to learn’; thus work to 
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promote AfL is expected to enhance L2L although this hypothesis needs to be tested and 
the relationships properly examined. 

• The development of new AfL/L2L practices need to be stimulated through some form of 
intervention because previous research indicates that they are not widespread in 
teachers’ current practice. 

• Intensive intervention by university people cannot be sustained in the long term and 
across whole systems therefore the intervention needs to be designed in accordance with 
a principle of parsimony. 

• Adaptation of the intervention is likely, and to be encouraged^, because the project has 
an interest in discovering and recording how teachers create new knowledge as they try 
out, and adapt, practices in their particular contexts i.e. in their schools, with their 
pupils, teaching an aspect of their subject. 

• Development and research have an iterative relationship because the research design 
needs to be sensitive to these adaptations in context in order to capture them. 

• In order to test portability of ideas and practices, implementation needs to be 
encouraged in a sufficient range of school contexts - urban/rural, large/small, mono- 
ethnic/multi-ethnic, elementary/high schools, and across different subject domains. 

• Supporting development work in a large sample of schools (n.43), even on a 
parsimonious scale (with an allocation of two days of an academic’s time per school per 
year) requires a large team, especially when this is added to research tasks. 

• A large project team provides an opportunity for the creation of a multidisciplinary 
group who learn from one another and deepen their research capacity even if none can 
claim to be a “compleat researcher” (Gorard, 2002). This acknowledges that in any 
community, knowledge is distributed in ways that are functional to its survival. 

• Implementation of project ideas over time needs to be monitored and described, 
requiring thick descriptions of practice in classrooms, of aspects of school policy, and of 
wider contextual influences that impact on classroom practice. 

• Qualitative data need to be combined with quantitative data so that comparisons across 
schools and over time, in relation to important features, can be facilitated. 

• Data need to be collected about defined outcomes, processes and contexts and the 
relationships among these explored. 

• Explanation needs to address the issue of causality although association should not be 
taken to imply a causal relationship, nor the direction of causality. 

• The activity of academics and researchers, and the differences among them in their 
approach, need to be taken into account through reflexivity. 

• A theorised audit trail approach offers the best strategy for generating plausible 
explanations, linking processes to outcomes, in the complex setting of this project. 

• Plausible rival explanations should also be deliberately sought to strengthen the 
scientific merit of project findings. 



* We wanted, of course, to avoid ‘lethal mutations’ (Brown, 1992). Our characterisation of the support from 
academics as ‘critical friendship’ to schools provided a means of moderating teachers adaptations by reference to 
principles derived from previous research. For example, teachers who wanted to interpret ‘sharing criteria of quality 
with students’ as encouragement to distribute model answers for reproduction in examinations, could be challenged 
to consider the importance of a learning orientation, rather than a performance orientation, if students are to learn 
how to learn. 
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• In a climate of increasing work intensification, participating schools and teachers need 
to be convinced that there are benefits to be gained from their collaboration in the 
project. These do not need to be in the form of assurances that test scores will rise but 
they do need to be convinced that their professional needs are recognised and supported, 
and that the work will be useful to them. 

In sum, certain principles for research design can be extracted from the assumptions outlined 
above: collaboration, intervention, parsimony, utility, relevance, adaptability, iteration, 
portability, scalability, sustainability, generalisability, plausibility, theoretical coherence, 
multidisciplinarity, reflexivity. 

A design experiment? 

We began work on the proposal for this project on 18‘^ November 1999^ We were bidding to be 
part of the Teaching and Learning Research Programme, which was set up specifically to 
respond to public concerns about educational research quality, so we were cognisant of the 
debates surrounding research design, including those in the US. However, we never explicitly 
described our proposal as an attempt to create a ‘design experiment’. Indeed the Design-Based 
Research Collective in the US was only forming at the same time (Kelly, 2003) although the 
work of Ann Brown(l992) was known on our side of the Atlantic. Now that the design 
experiment, as an approach to educational research in complex settings, has been more fully 
explored (see the special issue of Educational Researcher, 32(1) January/February 2003), the 
resonance between these ideas and our own attempts to construct at project, which will be both 
scientifically rigorous and practically useful, are remarkable. We would claim that it has all the 
characteristics of design experiments* described by Cobb, Conffey, diSessa, Lehrer and 
Schauble (2003). It is iterative, process-focussed, interventionist, collaborative, multileveled, 
utility oriented and theory drive. Our additional principles suggest that other characteristics may 
need to be considered also. 

We are puzzled however that some of the current debate implies that design experiments ‘rely on 
narrative accounts as data for modifying theory’ (Shavelson, Phillips, Towne and Feuer, 2003). 
Our experience suggests that through combining qualitative and quantitative approaches (though 
not necessarily taking the prototype back to the laboratory, as these writers suggest), it is 
possible to integrate narratives of the evolving design process with more traditional studies 
comparing outcomes, with some controls, to warrant a causal claim. Of course, we are still a long 
way from completing our project so whether we will be successful in this remains to be seen. (A 
further paper on this specific issue will be given at another symposium in this conference^.) 

Project Plan 

In so far as all projects within the TLRP have a principal aim to research ways of enhancing 
outcomes for learners'*’ , we needed to decide what outcomes are of interest and how we might 



’ It was more than a year later that we were informed that we had succeeded in the ESRC competition and could 
start work. 

® ‘Design studies’ according to those who object to this appropriation of the term from natural science. 

’ ‘Deepening capacity through innovative research design: researching learning how to learn in classrooms, schools 
and networks’ by Mary James, Paul Black and Robert McCormick, in the BERA Symposium, ‘The UK’s Teaching 
and Learning Research Programme; responding to challenges for educational research’ 24'*' April 2003. 

In our project, learners are construed as both students and teachers, and even schools as organisations. 
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measure them. We accepted that one measure would need to be students’ scholastic attainment as 
currently measured in schools through nationally prescribed key stage tests and examinations (at 
age 7, 11, 14 and 16). Participation and engagement as indicated by school attendance records 
are also relevant. However, our interest in learning how to learn as an outcome, as well as a 
process, led us to develop a dynamic assessment instrument. Although the nature of this 
instrument prevents us from using it on a wide sample in the current study, we will examine the 
relationship between some students’ performance on this measure and the standard measures of 
attainment that most children in England now undergo. We make the assumption that these 
outcomes are, at least in part, the result of classroom interactions with teachers, peers and other 
tools and artefacts, and that these classroom interactions and practices are influenced by 
teachers’ and students’ beliefs about learning. In the same way, we hypothesise that teachers’ 
and students’ individual and collective beliefs and practices are themselves outcomes and, at 
least in part, the result of their experience of professional development, school culture, 
management practices and networking opportunities both within and across schools. Through 
this process of backward-mapping we have constructed a ‘causal model’ with process and 
outcome links that we intend to investigate (see Figiorel). 

Figiore 2 provides more detail of the data we are collecting and the specific role these elements of 
data are expected to have in illuminating the impact of interventions on practice and outcomes. 
Thus we make a distinction between those data that are measures of antecedents and outcomes, 
and those that are principally indicators of mediating or contextual variables. In general, the 
former, derived from survey or test instruments, will render longitudinal (pre- and post- 
development) quantitative information both within and across schools. The latter, drawing on 
mainly narrative sources, will provide data that will be used to investigate the nature of links 
indicated by correlations. We intend to place most emphasis on comparisons between cases (e.g. 
different schools with different rates and patterns of development) but it may be possible to make 
some use of control schools, which have no contact with the project, by drawing a matched 
sample from a large national data-set available to Cambridge researchers. Case comparisons will 
be important because the ‘intervention’ is not tightly controlled. This has been quite deliberate 
because we are particularly interested in how schools and teachers take up, adapt and implement 
project ideas and what conditions promote effective practice and enhanced outcomes. 

In Figiore 2, we also distinguish between the locus and the focus of the intervention of the project 
team. Our primary intervention was in the form of an initial in-service session for a school or 
group of schools, followed by audit of practice and action-planning. The focus of this session 
was presentation and discussion of research and ideas for encouraging learning how to learn 
through assessment for learning". What happened next depended to an extent on schools’ own 
analysis of their needs although follow-up workshop activities, on: questioning, feedback, 
sharing criteria of quality, and peer- and self-assessment, were made available. These all focus 
on classroom practice. 

Feedback of the results of the staff questionnaire is the key intervention focusing on school 
policy and practice. This questionnaire has a dual purpose (see below for more detail). On the 
one hand it is designed to provide the research team with data of staff perceptions and values in 



" Development materials and other project resources are publicly available on our project website: 
http://leamtoleam.ac.uk 



7 




8 



relation to school assessment practice, professional learning, and management systems and 
practices. On the other hand, results can be used as diagnostic tools for schools in self- 
evaluation. Further materials for follow up activities are made available to schools to help them 
respond to the issues for school-level policy and practice that this instrument reveals. 

One of the principles underpinning the project strategy is that research instruments should be 
useful to teachers and schools as well as useful to researchers. Thus there is an intention that all 
instrumentation should be in a form that can be ‘left behind’ for teachers to use, alongside more 
familiar forms of in-service resources. In order to manage this, the role of the project website has 
developed in the expectation that it will become an expanding resource of materials, instruments 
and examples of practice that teachers can use and exchange. For this reason, the website is 
itself a kind of intervention, which principally operates at network level, having a role in 
knowledge creation and exchange between teachers and schools. 

Some of the parts of this plan were described in the project proposal but much of the detail has 
become clear only as the project has evolved in the iteration between development and research, 
and in the attempt to balance what might be methodologically desirable with what is practically 
feasible and manageable. We see this as appropriate, if not inevitable. As we arrive at the half- 
way mark in our time-scale we are reasonably confident that we have all the components for 
development and data collection in place, so we can turn more of our attention to analysis and 
theory development. Of course, our instrumentation was developed on the basis of existing 
theory, and the ‘substantive’ theory implicit in our causal model, which we intend to test and 
modify. But we also hope to build some new theory and to examine how our findings might 
illuminate, or be illuminated by , ‘formal’ theories such as activity theory (Engestrom, 1987), 
distributed cognitions theory (Salomon, 1993), or communities of practice theories (Wenger, 
1998). The final right-hand column of our plan (Figure 2) records some of the theoretical 
resources we expect to bring to this task. Given the fact that the team is made up of academics 
and researchers with different backgrounds we decided not to locate this study in a single narrow 
theoretical framework but to create a project which would enable us to test a number of 
analytical perspectives. We believe that the phenomena we are interested in can be viewed 
through a number of lenses that each can enrich and deepen our vision. 

In the remainder of this paper, we turn from the general design of our project to one particular 
aspect of it, in order to illustrate more clearly how one of our instruments has been developed 
and used to meet some of our scientific goals as well as to make a contribution to enhanced 
educational practice. 



The design and uses of the staff questionnaire 

The staff questionnaire is an example of a tool with both research and development functions. In 
this section, the development, design, preliminary findings and uses of the staff questionnaire are 
outlined as a way of exemplifying some of the issues involved when the service of both research 
and development are equally paramount within the design and applications of a single 
instrument. 
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Instrument development and underlying assumptions 

At the school level we are interested in developing understandings of the organisational 
conditions of schools that promote changes in teachers’ and students’ learning how to learn 
beliefs and practices. An important assumption that we make is that schools themselves need to 
develop the processes and practices of organisations that are learning how to learn if they are 
authentically to embody the conditions that, according to our theory, promote changes in 
teachers’ and students’ behaviours. To obtain comprehensive measures of such conditions, we 
need to go beyond questions of conditions that operate at the purely organisational level, such as 
school policies and the formulation of school improvement plans, and to incorporate a focus on 
teachers’ professional learning practices and their adoption of classroom practices that we 
theorise to promote learning how to learn knowledge, dispositions and practices among students. 
Reflecting an emphasis on learning how to learn processes at all levels of the school 
organisation, the staff questioimaire has been designed to provide antecedent, outcome and 
change measures of school climate and culture in relation to teachers’ classroom assessment 
practices (section A), teachers’ professional learning practices and beliefs (section B) and school 
management practices and systems (section C). 

Section A of the questionnaire consists of thirty statements relating to teachers’ classroom 
assessment practices. These statements reflect assumptions we make about the coherence 
between the concepts and practices of assessment for learning and learning how to learn. The 
statements in section A are based on a set of principles of practice developed by the Assessment 
Reform Group (2002) which build on the Black and Wiliam review (1998a). These principles 
can be summarised as a model of assessment practice in service to students’ needs and in 
recognition of the importance of their critical participation in the processes of their own learning, 
and in the plaiming of the next steps in their learning. Thus assessment for learning is seen as an 
integral part of effective plarming of teaching and learning and of routine classroom practice. 
Promoting understanding among students of learning objectives and criteria of assessment are 
viewed as important modes of critical learning engagement. Teachers arrange for the 
involvement of students in the processes of assessment. Students are encouraged to reflect on 
their own learning processes, their strengths and their weaknesses and to recognise the 
importance of both effort and error in making progress. 

Section B consists of 28 statements relating to teachers’ professional learning practices and 
beliefs. Statements in this section recognise the reciprocal importance between school 
development and the professional learning of teachers (Bradley, Cormer and Southworth, 1994; 
Hargreaves, 1994) and draw on a range of principles, which according to our theory, characterise 
teachers who themselves are learning how to learn. Informed by notions of reflective practice 
(e.g. Schon, 1987, Pollard and Taim, 1993), teachers who are learning how to learn sustain an 
engagement with iimovation and are disposed to reflect critically on their practice. We 
understand such reflective practices as incorporating both social and individual learning 
processes (e.g. Salomon, 1993). Teachers engage collaboratively with a wide range of sources 
and evidence, for example from peer observations, joint planning, experimentation and team 
teaching with colleagues, from consultations with students, and from research reports and web- 
based resources. They reflect on and modify their classroom practice in ways that relate to such 
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different kinds of available evidence. These kinds of collaborative learning processes are 
fostered in a culture of support and trust in which teachers’ inter-personal capacity (Mitchell and 
Sackney, 2000), to deepen the bonds of learning with colleagues, are further enhanced. Teachers 
who are learning how to learn are not only models of learning how to learn for their students, but 
are also committed to and confident in the development by their students of learning how to learn 
knowledge, skills and practices. 

Section C consists of 26 statements relating to school management systems and practices. 
Statements in this section reflect a range of frameworks as follows: (i) conditions for school 
improvement, developed as part of the ‘Improving the Quality of Education for All’ project 
(Ainscow, Hopkins, Southworth and West, 1998), (ii) principles of organisational learning how 
to learn that emphasise the importance of double loop learning (Argyris and Schdn, 1978), 
school self-evaluation (MacBeath, 1999), and the importance to organisational growth of 
increases in personal, interpersonal and organisational capacities (Mitchell and Sackney, 2000) 
and (iii) the central role of knowledge creation and management in schools to the successful 
promotion of innovation and change (Hargreaves, 1 999). 

The importance of inclusive and self-critical processes of decision-making is recognised in a 
widening of participation in decision-making processes through open communication combined 
with the fostering among teachers of critically reflexive skills to question the underlying values 
of established modes of school decision-making. Instrumental research and inquiry skills are also 
important facets of the school as a learning how to learn organisation. Thus acquisition by 
schools of skills and methods of systematic research and inquiry as a means of generating 
relevant data to inform organisational level decision-making is a key dimension of a school’s 
reflexivity. In taking seriously the expertise of their staff, schools that learn how to learn use 
professional knowledge audits about specific aspects of expertise as a way of mapping what staff 
know about teaching and learning. This acknowledges to staff the value of their learning and 
knowledge to school planning and development. A final assumption about schools as learning 
how to learn organisations refers to their recognition of the importance to their learning of intra- 
and inter-school networking as a process of promoting knowledge creation within and across 
schools. Schools optimise knowledge transfer within the school and seek to gain access to 
knowledge from and exchange knowledge with other schools through developing the skills and 
readiness to learn through networking and to promote networking as a mode of learning among 
staff 

Design 

The staff questionnaire has a double scale structure throughout the three main sections. In this 
way it has been shaped by assumptions similar to those that informed the design of the ‘teacher 
questionnaire’ used in the Improving School Effectiveness project - ISEP - (MacBeath and 
Mortimore, 2001). The use of two likert-type scales throughout the questionnaire is a usefixl 
means of eliciting staff perceptions of current practices (scale X) at their school, on the one 
hand, and their values (scale Y) regarding the importance of different practices and beliefs for 
creating opportunities for students to leam, on the other. ‘Where the school is now’ {current 
practice) and ‘Where staff would like the school to be’ {their values) are questions that form the 
basis for indicating gaps between current practice and values at the school as they relate to 
teachers’ classroom practices, their professional learning, and management systems and 
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practices. Feeding back questionnaire data to individual schools so that they can identify a range 
of patterns, including gaps between the values and practices of staff, represents an important 
school level developmental intervention. This intervention is mediated by the critical friend 
allocated to each of the participating schools (see below). 

In thinking through the design and uses of the questionnaire we were influenced by the work in 
ISEP of Robertson, Sammons, Thomas and Mortimore (2001:44). The use of the dual scale 
structure and the strategy of feeding back data to schools individually reflects principles of 
organisational learning and development based on models of self-evaluation (MacBeath, 1999) 
where the emphasis rests on the needs of individual schools, continuous review and self 
correction (Dalin and Rust, 1996), systematic self-assessment strategies and instruments, and the 
use of external consultants or critical friends. 

Analysis 

Discussion of the staff questionnaire here is based on the responses of a sub-sample consisting of 
779 members of staff from 20 schools (1 1 primary and 9 secondary). These data were collected 
between September and November 2002 after trialling the instrument with 370 staff at 9 schools 
between March and June 2002. 

For preliminary analytical and data feedback purposes three broad distinctions were made 
between staff at different levels of the school organisation as follows; (i) learning support staff, 
(ii) teachers with little or no managerial responsibilities, and (iii) middle and senior management 
staff Our first analytic task was to disentangle the complexity of the data generated by responses 
to 176 questionnaire items. Clusters of variables were developed from a combination of 
statistically generated factors based on responses to staffs perceptions of current practices (scale 
X on the questionnaire) and theoretical groupings based on the conceptualisation that informed 
the development of the questionnaire. Factor analyses were carried out for each section of the 
questionnaire separately for managers on the one hand and teachers and learning support staff 
on the other. High correspondence was found for the factor structures of both groups of staff' 

The reliability analysis showed a generally high level of reliability based on Cronbach’s alpha 
for the factors identified for all three sections. Reliability coefficients for the three factors in 
section A were.8634, .4736 & .5313; for the four factors in section B were .8614, .7401, .6746, 

& .6200; for the four factors in section C were .8998, .8847, .8238 & .7520. Full discussion of 
this analysis will appear in a separate paper. We found high correspondence between the factor 
structure derived from our statistical analyses and the groupings developed from our theoretical 
frameworks in all three sections. Following an iterative process of comparing results from the 
factor analysis with our theoretical assumptions we arrived at a set of groupings of variables as 
follows. The list of factors derived from the statistical analysis combined with our theoretical 
formulations are listed below. Descriptions of each factor in terms of their component 
questionnaire items appear in Appendix 1 . 



The decision to combine teachers’ and learning support staffs responses was due to an inadequate sub-sample of 
learning support staff at the time of the analysis. 

The statistical factors that we identified were based on staff responses to questionnaire items on the X scale 
(current practice). 
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Classroom assessment practices (section A) 

Factor 1 : Making learning explicit 

Factor 2: Promoting learning autonomy 

Factor 3: Teacher-centred, performance orientation 

Teachers 'professional learning practices and beliefs (section B) 

Factor la: Inquiry: using different sources of knowledge 

Factor lb: Inquiry: reflection and change 

Factor 2: Collective learning 

Factor 3: Mutual support and reassurance 

Factor 4: Valuing learning 



School management systems and practices (section C) 
Factor 1 : Deciding and acting together 
Factor 2: Developing a sense of where we are going 
Factor 3: Inquiry 

Factor 4: Supporting professional development 



Uses of the questionnaire data 

The primary research purposes of the data analysis is to describe trends in the data across staff at 
different levels of the school organisation and across different schools in the sample*'*. The factor 
structure that we have developed forms the basis for the main part of this analysis. And yet there 
are clear dilemmas in using such survey instruments. Like the ISEP team before us (Robertson, 
Sammons, Thomas and Mortimore, 2001 :44), we recognise the tension in designing a research 
instrument capable of being used reliably across a range of schools but also amenable to the 
provision of feedback which is relevant to the context and needs of each of the schools in our 
sample. 

The development task is to arrive at a number of different descriptive representations of the data 
that cater to the different questions and preferences that schools might have. Thus we prepared 
data reports for each school that compared responses of managers, teachers and LSAs. Data fforh 
the rest of the sub sample were included to provide an optional basis of comparison (see 
Appendix 2 for a small selection of examples). Raw data tables and bar charts present 
questionnaire item responses separately under each of the three sections of the questionnaire and 
listed under the headings of the factors identified for each section. From these tables, schools can 
then read the distribution of responses for any item for managers, teachers or learning support 
staff, compare mean responses and compare differences between the mean practice-value 
responses. Rank order tables were also developed to enable schools to identify which practices 
have the highest and lowest practice-value gap. Grouped data in the form of standardised scores 
for each factor were used to show distributions of scores for the three broad categories of staff in 
the form of histograms and reverse cumulative percentage charts. 



We plan a second administration of this questionnaire at the end of the project to analyse change over time, both 
for the sample as a whole and for each school. 
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The feedback of data in these forms provide school management teams with a range of different 
perspectives for ‘seeing’ their own individualised data sets. How they respond to the data carries 
important implications for both research and development dimensions of our study. Understood 
as a strategy for change, the data can feed into a school’s improvement planning, policy and staff 
development. Where the data are understood as an opportunity for a school to learn more about 
itself, opportunities arise for researching how schools make use of such data. MacBeath (2001 :3- 
4) identifies three levels of school response to questionnaire data that reflect authentic, double 
loop self-critical learning that we assume to underpin organisational learning how to learn: 

At the technical level a school might ask of itself - What did we learn about design, data, 
data gathering, validity, the nature of ‘data’, the context in which it is gathered, and about 
alternative ways in which we may gather data in the future? At the cultural level a school 
may ask - What did we learn about my/our responses? What are our defences? How 
habitual, and habit-forming are these? How deeply are these embedded in the way we do 
things round here? [At the structural level we may ask] - What have we learned about the 
structures within which we gather and use data? What is there about those structures that 
facilitate and inhibit the gathering and use of data in a way that enhances our capacity to 
learn? 

To the extent that we are able to research how schools make sense of and use the data that we 
develop, the developmental application of the staff questionnaire data can be recognised as a 
powerful opportunity to study the processes by which schools step out of the single learning loop 
and enter the less certain and riskier double loop of learning how to learn. 



Feedback of data to schools by ‘critical friends’ 

Feeding back the data from the staff questionnaire is undertaken by the school’s ‘critical friend’: 
one of the university academics attached to the school. Before taking the data back to the 
schools, the critical friends themselves needed to be familiar with them and confident in handling 
and explaining them. Team members shared their experiences and insights of feeding back data 
in similar contexts. On this basis we developed a guide to reading the data that could both be 
used by critical friends and left behind in the school. However, the guide was regarded as a 
‘prop’ and not a prescription. Critical friends were encouraged to adapt it if they wished and to 
take school colleagues through the data in which ever way seemed most appropriate. 

Although critical friends are only able to allocate two days of direct contact to each school per 
year, most had some contact with the schools in the initial stages of the project, usually through 
leading professional development sessions. So, when the time came to feed back the 
questionnaire data a relationship had begun to be established between the critical fiiend and 
senior school colleagues. Critical friends were beginning to know the school, and, hopefully, 
school colleagues had developed a degree of trust in their critical friend. Trust, understanding 
and providing data are elements of Costa and Kallick’s (1993) description of a critical friend. 

The design of the staff questionnaire was such that it revealed school managers’, teachers’ and 
learning support assistants’ views about assessment practices, teachers’ professional learning. 
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and perhaps most sensitively, school management practices and systems. For this reason critical 
friends fed back the data to a small group of senior staff first, generally the school principal and 
school project co-ordinator, before it was decided how to share the data more widely. 

To help school colleagues understand and come to terms with the data, critical friends typically 
started by making reference to the format of the staff questionnaire, with its three sections and X 
and Y scales. In the current UK scene where schools now receive large quantities of performance 
data from both national and local government, staff are becoming more and more sophisticated 
in their understanding and use of quantitative data. However, depth of understanding still varies 
between individuals and schools. Critical friends therefore needed to gauge staff familiarity with 
data presented in different forms, and adapt their presentations accordingly. They also had to be 
prepared for a variety of possible reactions to the data, including denial of the validity of the data 
and demands to know how individual members of staff had completed the questionnaire. In 
practice these reactions were rare but critical friends needed to be prepared for them and to 
respond appropriately. 

Once school colleagues were comfortable with the presentation of the results of the staff 
questionnaire, they then began to explore the messages contained in the analysis. For example, 
the critical friend may have pointed out instances of large discrepancies between the responses of 
different groups’ and initiated a discussion about the possible meanings of those discrepancies. 
Often, however, schools were keen to take the data away, examine them themselves, and take the 
necessary time to identify and reflect on the issues raised. The nature of the intervention by the 
critical friend then changed, from explaining the origin and format of data, to asking provocative 
questions (another element in Costa and Kallick’s description of a critical friend) about the data 
the school has chosen to focus upon, about the way they interpreted them, and about their 
proposed course of action. 

The L2L team also developed a pack of materials, or a ‘toolbox’ of instruments and activities, to 
support further inquiry into the issues emerging from the analysis. Some tools relate directly to 
the questionnaire, for example providing a structure for reflecting upon the data, whilst others 
support development in an aspect of school practice that may have been identified through the 
questionnaire (see the ‘focus of intervention’ column in Figure 2). The critical friend may have 
suggested particular tools to use or adapt, thus initiating another form of intervention at school 
level. 

All these choices and actions are logged, either by the critical friend or by the school co- 
ordinator, or by both, on a restricted access part of the project website, thus providing narrative 
data for the interpretation of change in beliefs, practices and outcomes over time. Thus, as the 
research feeds into development, so evidence of development feeds back into the research in an 
iterative cycle. 



Challenges 

We continue to be excited by this opportunity to be involved in a ground-breaking Project within 
a unique research Programme. Although the challenges are very great, and not a little daunting. 



we believe that we should be prepared to take risks and try to work with colleagues to pioneer 
new designs for educational research that will be both practice-enhancing and scientifically 
credible. There are, of course, a number of threats to the achievement of these goals. Some 
derive from resource constraints, others from issues surrounding the management of large project 
teams, with inevitable personnel changes during a four year timescale, others from the political 
climate which has, at times, been hostile to educational research, or doubtful of its capacity to 
produce tmstworthy and useful findings. 

Another challenge derives from the shifting contexts in which we work. We cannot hold the 
educational world still while we examine it, so we have to create designs that take shifting 
contexts into account. This is very challenging, especially when different groups of users make 
demands for different kinds of knowledge. We agree with David Berliner (2002) that educational 
research is the ‘hardest science of all’. We know that there are areas in which we will fail but we 
are also certain that we will learn a great deal from this project and have something valuable to 
contribute both to ‘science’ and to practice. 



References 

Ainscow, M., Hopkins, D., Southworth, G. and West, M. (1998) Creating the Conditions for 
School Improvement. London: David Fulton. 

Argyris, C. and Schon, D. (1978j Organisational Learning : A Theory of Action Perspective. 
Reading, MA: Addison Wesley. 

Assessment Reform Group (1999) Assessment for Learning: Beyond the Black Box. Cambridge: 
Cambridge University School of Education. 

Assessment Reform Group (2002) Assessment for Learning: 10 Principles. Cambridge: 
Cambridge University Faculty of Education. 

Berliner, D. (2002) Educational Research: The Hardest Science of All. Educational Researcher, 
31(8): 18-20. 

Black, P. J. & Wiliam, D. (1998a). Assessment and classroom learning. Assessment in 
Education: Principles Policy and Practice, 5(1), 7-73. 

Black, P. and Wiliam, D (1998b) Inside the Black Box London: King’s College School of 
Education 

Black, P., Harrison, C., Lee, C., Marshall, B. & Wiliam, D. (2002) Working Inside the Black 
Box: Assessment for Learning in the Classroom, London: King’s College School of Education. 

Bradley, H., Conner, C. and Southworth, G. (eds.) (1994) Developing Teachers, Developing 
School: making inset effective for the school, London: David Fulton. 



15 




16 



Brown, A. (1992) Design experiments: theoretical and methodological challenges in creating 
complex interventions in classroom settings, The Journal of the Learning Sciences, 2(2); 141- 
178. 

Cobb, P., Confrey, J., diSessa, A., Lehrer, R. & Schauble, L. (2003) Design Experiments in 
Educational Research, Educational Researcher, 32(1); 9-13. 

Costa, A. and Kallick, B. (1993) Through the Lens of a Critical Friend. Educational 
Leadership. 51(2)49-51. 

Crooks, T.J. (1988) The Impact of Classroom Evaluation Practices on Students, Review of 
Educational Research. 58(4); 438-81. 

Dalin, P and Rust, V. (1996) Towards Schooling for the Twenty-First Century. London: Cassell. 

Engestrdm, Y. (1987). Learning by Expanding: An Activity-theoretical Approach to 
Developmental Research.. Helsinki: Orienta-Konsultit. 

Frost, D., Durrant, J., Head, M. & Holden, G. (2000) Teacher-led School Improvement. London: 
Routledge/Falmer. 

Gorard, S. (2002) How do we overcome the methodological schism (or can there be a ‘compleat 
researcher)? Occasional Paper Series; Paper 47, Cardiff; RCBN, Cardiff University. 

Hargreaves, D. (1994) ‘The New Professionalism; The Synthesis of Professional and 
Institutional Development’, Teaching and Teacher Education, Vol. 10, No. 4, pp. 423-438. 

Hargreaves, D. (1999) ‘The knowledge Creating School’, British Journal of Educational 
Research, Vol. 47, No. 2, pp. 122-144. 

James, M. (1998) Using Assessment for School Improvement. Oxford; Heinemann. 

Kelly. A. E. (2003) Research as Design, Educational Researcher, 32(1); 3-4. 

MacBeath, J. (1999) Schools Must Speak for Themselves. London: Routledge. 

MacBeath, J. (200 Ij Can Schools Learn how to Learn? From Single to Double Loop. Internal 
project paper for learning How to Learn project. 

MacBeath, J. and Mortimer, P (eds.) (2001) Improving School Effectiveness. Buckingham; Open 
University Press. 

McCormick, R. and James, M. (1983, 1988) Curriculum Evaluation in Schools, London: 
Routledge. 



16 




17 



Mitchell, C. and Sackney, L. (2000) Profound Improvement - Building capacity for a Learning 
Community. Lisse: Swets and Zeitlinger. 



National Research Council (2002) Scientific Research in Education. In R. J. Shavelson & L. 
Towne (eds.) Committee on Scientific Principles for Educational Research, Washington, DC: 
National Academy Press. 

Pollard, A and Tann, S. (1993) Reflective Teaching in the Primary School. London: Cassell. 

Robertson, P., Sammons, P., Thomas, S., and Mortimer, P. (2001) ‘The Research Design and 
Methods’, in J. MacBeath and P. Mortimer (eds.) Improving School Effectiveness. Buckingham: 
Open University Press. 

Salomon, G. (1993) ‘No Distribution Without Individual’s Cognition: A Dynamic Interactional 
View’, in G. Salomon (ed.) Distributed Cognitions: Psychological and Educational 
Considerations. Cambridge: Cambridge University Press. 

Schon, D. (1987) Educating the Reflective Practitioner. San Francisco: Jossey-Bass. 

Shavelson, R.J., Phillips, D.C., Towne, L. & Feuer, M.J. (2003) On the Science of Education 
Design Studies, Educational Researcher, 32(1): 25-28. 

Stokes, D.E. (1997) Pasteur’s Quadrant: Basic science and technological innovation. 
Washington, DC: Brookings Institution Press. 

Wenger, E. (1998) Communities of Practice: learning, meaning and identity. Cambridge, 
Cambridge University Press. 



17 



ERIC 
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Figure 2: L2L Development and Research Plan 
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Appendix 1: Definition of factors in terms of component questionnaire items 
Section A: Classroom Assessment Practices 

Factor 1: Making Learning Explicit 



Questionnaire 

items 


Questionnaire statements 


1 


Assessment provides teachers with useful evidence of students’ understandings 
which they use to plan subsequent lessons 


4 


The feedback that students receive helps them improve. 


10 


Students are told how well they have done in relation to their own previous 
performance. 


11 


Students’ learning objectives are discussed with students in ways they 
understand. 


14 


Teachers identify students’ strengths and advise them on how to develop them 
further. 


15 


Students are helped to find ways of addressing problems they have in their 
learning. 


16 


Students are encouraged to view mistakes as valuable learning opportunities. 


17 


Students are helped to think about how they learn best. 


18 


Teachers use questions mainly to elicit reasons and explanations from their 
students. 


20 


Students’ errors are valued for the insights they reveal about how students are 
thinking. 


21 


Students are helped to understand the learning purposes of each lesson or series 
of lessons. 


22 


Assessment of students’ work is mainly in the form of comments. 


26 


Students are helped to plan the next steps in their learning. 


27 


Pupil effort is seen as important when assessing their learning. 


28 


Assessment criteria are discussed with students in ways that they understand. 


30 


Teachers regularly discuss with students ways of improving learning how to 
learn. 
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Section A: Classroom Assessment Practices 



Factor 2: Promoting Learning Autonomy 



Questionnaire 

items 


Questionnaire statements 


6 


Students are given opportunities to decide their own learning objectives. 


9 


Teachers’ assessment practices help students to learn independently. 


13 


Teachers provide guidance to help students assess their own work. 


19 


Teachers provide guidance to help students to assess one another’s work. 


24 


Teachers provide guidance to help students assess their own learning. 


29 


Students are given opportunities to assess one another’s work. 



Factor 3: Teacher-centred, Performance-orientation 



Questionnaire 

items 


Questionnaire statements 


2 


The next lesson is determined more by the prescribed curriculum than by how 
well students did in the last lesson. 


3 


The main emphasis in teachers’ assessments is on whether students know, 
understand or can do prescribed elements of the curriculum. 


5 


Students are told how well they have done in relation to others in the class. 


7 


Teachers use questions mainly to elicit factual knowledge from their students. 


8 


Teachers consider the most worthwhile assessment to be assessment which is 
undertaken by the teacher. 


12 


Assessment of students’ work consists primarily of marks and grades. 


23 


Students’ learning objectives are determined mainly by the prescribed 
curriculum. 






Section B: Professional Learning Practices and Beliefs 

Factor la: Inquiry: Using Different Sources of Knowledge 



Questionnaire 

items 


Questionnaire statements 


2 


Staff draw on good practice from other schools as a means to further 
their own professional development. 


3 


Staff read research reports as one source of useful ideas for improving 
their practice. 


4 


Staff use the web as one source of useful ideas for improving their 
practice. 


5 


Students are consulted about how they learn most effectively. 


6 


Staff relate what works in their own practice to research findings. 



Factor lb: Inquiry: Reflection and Change 



Questionnaire 

items 


Questionnaire statements 


7 


Staff are able to see how practices that work in one context might be 
adapted to other contexts. 


9 


Staff reflect on their practice as a way of identifying professional learning 
needs. 


10 


Staff experiment with their practice as a conscious strategy for improving 
classroom teaching and learning. 


11 


Staff modify their practice in the light of feedback from their students. 


12 


Staff modify their practice in the light of published research evidence. 


13 


Staff modify their practice in the light of evidence form self-evaluations of 
their classroom practice. 


14 


Staff modify their practice in the light of evidence from evaluations of their 
classroom practice by managers or other colleagues. 






Section B: Professional Learning 



Factor 2: Collective Learning 



Questionnaire 

items 


Questionnaire statements 


15 


Staff carry out joint research/ evaluation with one or more colleagues as 
a way of improving their practice. 


16 


Staff regularly collaborate to plan their teaching. 


17 


Staff regularly observe each other in the classroom and give each other 
feedback. 


18 


Staff engage in team teaching as a way of improving practice. 


21 


Teachers make collective agreements to test out new ideas. 


22 


Teachers discuss openly with colleagues what and how they are 
learning. 


23 


Staff frequently use informal opportunities to discuss how students 
learn. 


28 


Staff discuss with colleagues how students might be helped to learn 
how to learn. 



Factor 3: Mutual Support and Reassurance 



Questionnaire 

items 


Questionnaire statements 


19 


If staff have a problem with their teaching they usually turn to 
colleagues for help. 


20 


Teachers suggest ideas or approaches for colleagues to try out in class. 


24 


Staff offer one another reassurance and support. 







Section B: Professional Learning 

Factor 4: Valuing Learning 



Questionnaire 

items 


Questionnaire statements 


1 


Staff as well as students leam in this school. 


25 


Staff believe that all students are capable of learning. 


26 


Students in this school enjoy learning. 


27 


Pupil success is regularly celebrated. 




Section C: School Management Practices and Systems 



Factor 1: Deciding and Acting Together 



Questionnaire 

items 


Questionnaire statements 


5 


There are processes for involving all staff in decision-making. 


6 


Teachers' professional know-how is used in the formulation of school 
policy and goals. 


7 


Teachers' professional know-how is used in the formulation of school 
policy, even where this leads to a questioning of established rules, 
procedures and practices. 


8 


Opporhinities are provided for teachers to critically evaluate school 
policy. 


9 


Staff are actively involved in evaluating school policy. 


10 


Staff participate in important decision-making. 


11 


There are processes for involving students in decision-making. 


B8 


Staff use insights from their professional learning to feed in to school 
policy development. 
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Section C: School Management Practices and Systems 

Factor 2: Developing a Sense of Where we are Going 



Questionnaire 

items 


Questionnaire statements 


1 


Senior management communicates a clear vision of where the school is 
going. 


2 


Staff have a commitment to the whole school as well as to their 
department, key stage and/or year group. 


3 


Senior management promotes commitment among staff to the whole 
school as well as to the department, key stage and/ or year group. 


4 


There is effective communication between senior management and 
teachers. 


12 


Staff have a good working knowledge of the School Development Plan. 


13 


Staff see the School Development Plan as relevant and useful to 
learning and teaching. 


14 


Staff development time is used effectively to realise School 
Development Plan priorities. 


15 


Staff development time is used effectively in the school. 







Section C: School Management Practices and Systems 

Factor 3: Inquiry 



Questionnaire 


Questionnaire statements 


items 




21 


Management supports teachers in sharing practice with other schools 
through networking. 


22 


Information is collected from teachers on those aspects of their work 
that they themselves think they do effectively. 


23 


Information is collected from teachers on effective ways they promote 
learning to learn skills and knowledge among students. 


24 


Information is collected from teachers on informal teacher networking 
in which they play an active role. 


25 


Teacher-initiated networking is an integral element of staff 
development. 


26 


Learning how to learn is an issue discussed in staff development time. 



Factor 4: Supporting Professional Development 



Questionnaire 


Questionnaire statements 


items 




16 


The school provides cover to allow staff joint panning time. 


17 


Teachers are encouraged to experiment with new ideas as a way of promoting 
professional growth. 


18 


Formal training provides opportunities for teachers to develop professionally. 


19 


Teachers are helped to develop skills to assess students’ work in ways that 
move their students on in their learning. 


20 


Teachers are helped to develop skills to observe learning as it happens in the 
classroom. 
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